Replace JSON File Storage with Knex.js ORM

Summary

Replaces the flat JSON file storage layer with Knex.js, enabling the ISMS Builder to run against SQLite, PostgreSQL, and MariaDB/MySQL backends while keeping the original JSON mode as a zero-config default.

Motivation

• Data integrity — JSON files are prone to corruption under concurrent writes and offer no transactional guarantees.
• Scalability — PostgreSQL/MariaDB support is needed for multi-user production deployments.
• Query power — SQL backends enable filtering, pagination, and aggregation that would be expensive on flat files.
• Operations — Standard DB tooling (backups, replication, monitoring) becomes available.

What Changed

New: Centralized Database Module (server/db/knexDatabase.js)

• Singleton Knex instance with promise-based init() / destroy()
• Auto-creates tables on first run (initSchema)
• Auto-adds missing columns on startup (ensureColumns) — safe for incremental deploys
• STORAGE_BACKEND env var selects the driver (json → original, sqlite → better-sqlite3, postgres → pg, mariadb → mysql2)

New: Knex-based Store Modules (server/db/stores/*.js)

21 new store modules that mirror the existing JSON store API but execute against Knex:
templateStore · ackStore · assetStore · auditStore · bcmStore · customListsStore · entityStore · findingStore · gdprStore · goalsStore · governanceStore · guidanceStore · legalStore · orgSettingsStore · orgUnitStore · publicIncidentStore · riskStore · soaStore · supplierStore · trainingStore

Modified: Routes & Storage Layer

All 22 route modules updated to await the async store calls. The server/storage.js facade now delegates to Knex stores when STORAGE_BACKEND !== 'json'.

Modified: Existing Store Shims (server/db/*.js)

The original JSON stores now re-export from server/db/stores/ to preserve backward compatibility for any direct imports.

New: Migration Tool (tools/migrate-json-to-knex.js)

Idempotent CLI script that reads all data/*.json files and inserts them into the configured Knex backend. Safe to re-run — skips existing rows by primary key.

STORAGE_BACKEND=sqlite  node tools/migrate-json-to-knex.js
STORAGE_BACKEND=postgres DB_HOST=... node tools/migrate-json-to-knex.js

Database-specific Fixes

• MariaDB: TEXT columns automatically promoted to MEDIUMTEXT to support large document bodies (e.g. guidance content with embedded OpenAPI specs).
• PostgreSQL: Uses INTEGER GENERATED ALWAYS AS IDENTITY for auto-increment primary keys.
• Acknowledgements route: getMode() and distribution creation made fully async.

Configuration

Testing

• JSON backend: 8/8 test suites pass (43 tests) — no regressions
• SQLite backend: 8/8 test suites pass
• PostgreSQL 16: 8/8 test suites pass (tested via Docker)
• MariaDB 11: 8/8 test suites pass (tested via Docker)
• Migration tool: Verified idempotent import of all 20+ JSON data files against SQLite and PostgreSQL

Files Changed

68 files: +6 527 / −743 lines

• 22 new — server/db/knexDatabase.js, server/db/stores/*.js, tools/migrate-json-to-knex.js
• 46 modified — route handlers, existing store shims, tests, storage.js

Breaking Changes

None. The JSON storage backend remains the default and is fully backward-compatible.

Deployment Notes

1. Deploy the new code — JSON mode continues to work unchanged.
2. When ready, set STORAGE_BACKEND=sqlite (or postgres/mariadb) and run the migration script.
3. Verify data integrity, then switch traffic to the new backend.